Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VL][GLUTEN-8020] Remove the libhdfs3 installation script required for static linking #8013

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[VL][GLUTEN-8020] Remove the libhdfs3 installation script required for static linking #8013

wants to merge 1 commit into from
+7 −45

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Nov 21, 2024 •
edited
Loading

What changes were proposed in this pull request?

We will only retain the dynamic libraries libhdfs.so or libhdfs3.so at runtime based on this benchmark here. So there is no need to keep the libhdfs3 installation script required for static linking.

How was this patch tested?

Existing compile scripts.

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@PHILO-HE
Copy link
Contributor

@JkSelf, can we also remove ENABLE_HDFS option from some build scripts?

@zhouyuan
Copy link
Contributor

@JkSelf could you please help to link with the right Issue?

@JkSelf JkSelf changed the title Clean up the libhdfs3 install [GLUTEN-8020] Clean up the libhdfs3 install Nov 22, 2024
@JkSelf JkSelf changed the title [GLUTEN-8020] Clean up the libhdfs3 install [GLUTEN-8020] Remove the libhdfs3 installation script required for static linking Nov 22, 2024
@github-actions GitHub Actions
Copy link

#8020

@PHILO-HE
Copy link
Contributor

@JkSelf, can we also remove ENABLE_HDFS option from some build scripts?

Seems we still need it for deciding whether to register hdfs filesystem.

@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 22, 2024

@JkSelf, can we also remove ENABLE_HDFS option from some build scripts?

Seems we still need it for deciding whether to register hdfs filesystem.

Yes. We also need to pass the enable_hdfs to velox compile scripts.

@FelixYBW
Copy link
Contributor

@JkSelf is document updated?

@FelixYBW FelixYBW changed the title [GLUTEN-8020] Remove the libhdfs3 installation script required for static linking [VL][GLUTEN-8020] Remove the libhdfs3 installation script required for static linking Nov 23, 2024
@yabinma
Copy link
Contributor

yabinma commented Nov 26, 2024

The changes is helpful for #7977 , libhdfs3 compile issue found during the test. I will bypass the libhdfs3.

@yabinma yabinma mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
BUILD VELOX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@JkSelf @PHILO-HE @zhouyuan @FelixYBW @yabinma